Full modification coverage through automatic similarity-based test case selection

Context: This paper presents the similarity approach for regression testing (SART), where a similarity-based test case selection technique is used in a model-based testing process to provide selection of test cases exercising modified parts of a specification model. Unlike other model-based regression testing techniques, SART relies on similarity analysis among test cases to identify modifications, instead of comparing models, hence reducing the dependency on specific types of model. Objective: To present convincing evidence of the usage of similarity measures for modification-traversing test case selection. Method: We investigate SART in a case study and an experiment. The case study uses artifacts from industry and should be seen as a sanity check of SART, while the experiment focuses on gaining statistical power through the generation of synthetical models in order to provide convincing evidence of SART’s effectiveness. Through posthoc analysis we obtain p-values and effect sizes to observe statistically significant differences between treatments with respect to transition and modification coverage. Results: The case study with industrial artifacts revealed that SART is able to uncover the same number of defects as known similarity-based test case selection techniques. In turn, the experiment shows that SART, unlike the other investigated techniques, presents 100% modification coverage. In addition, all techniques covered a similar percentage of model transitions. Conclusions: In summary, not only does SART provide transition and defect coverage equal to known STCS techniques, but it exceeds greatly in covering modified parts of the specification model, being a suitable candidate for model-based regression testing.